Diffusion Networks, Products of Experts, and Factor Analysis

Hinton (in press) recently proposed a learning algorithm called contrastive divergence learning for a class of probabilistic models called product of experts (PoE). Whereas in standard mixture models the “beliefs” of individual experts are averaged, in PoEs the “beliefs” are multiplied together and then renormalized. One advantage of this approach is that the combined beliefs can be much sharper than the individual beliefs of each expert. It has been shown that a restricted version of the Boltzmann machine, in which there are no lateral connections between hidden units or between observation units, is a PoE. In this paper we generalize these results to diffusion networks, a continuous-time, continuous-state version of the Boltzmann machine. We show that when the unit activation functions are linear, this PoE architecture is equivalent to a factor analyzer. This result suggests novel non-linear generalizations of factor analysis and independent component analysis that could be implemented using interactive neural circuitry.